6  Chi square test

6.1 Preparation

• Load packages:

library("readxl")
library("tidyverse")

• Load the data sets:

data <- read_xlsx("Paquot_Larsson_2020_data.xlsx")

6.1.1 The \(\chi^2\)-test

The next step is to compute a test statistic that indicates how strongly our data conforms to \(H_0\). For instance, the Pearson \(\chi^2\) statistic is commonly used for categorical variables. It requires two types of values: the observed frequencies \(n_{ij}\) in our data set and the expected frequencies \(m_{ij}\), which we would expect to see if \(H_0\) was true.

This table represents a generic contingency table where \(X\) and \(Y\) are categorical variables. Each \(x_i\) represents a category of \(X\) and each \(y_j\) represents a category of \(Y\). In the table, each cell indicates the count of observation \(n_{ij}\) corresponding to the \(i\)-th row and \(j\)-th column.

			\(Y\)
		\(y_1\)	\(y_2\)	…	\(y_J\)
	\(x_1\)	\(n_{11}\)	\(n_{12}\)	…	\(n_{1J}\)
	\(x_2\)	\(n_{21}\)	\(n_{22}\)	…	\(n_{2J}\)
\(X\)	…	…	…	…	…
	\(x_I\)	\(n_{I1}\)	\(n_{I2}\)	…	\(n_{3J}\)

# Cross-tabulate the frequencies for the variables of interest

freqs <- table(data$ORDER, data$SUBORDTYPE); freqs

       
        caus temp
  mc-sc  184   91
  sc-mc   15  113

We calculate the expected frequencies by using the formula

\[ m_{ij} = \frac{i\textrm{th row sum} \times j \textrm{th column sum}}{\textrm{number of observations}}. \]

# Compute expected frequencies

## Calculate row totals
row_totals <- rowSums(freqs)

## Calculate column totals
col_totals <- colSums(freqs)

## Total number of observations
total_obs <- sum(freqs)

## Calculate expected frequencies
expected_table <- outer(row_totals, col_totals) / total_obs

expected_table

           caus      temp
mc-sc 135.79404 139.20596
sc-mc  63.20596  64.79404

Definition of the \(\chi^2\)-test

Given a sample with \(n\) observations and \(k\) degrees of freedom (\(df\))¹, the \(\chi^2\)-statistic measures how much the observed frequencies deviate from the expected frequencies for each cell in a contingency table (cf. Heumann, Schomaker, and Shalabh 2022: 249-251):

\[ \chi^2 = \sum_{i=1}^{I}\sum_{i=j}^{J}{\frac{(n_{ij} - m_{ij})^2}{m_{ij}}}. \] The test stipulates that …

1. all observations are independent of each other,
2. 80% of the expected frequencies are \(\geq\) 5, and
3. all observed frequencies are \(\geq\) 1.

Implementation in R: Manual vs. automatic

# Compute chi-squared scores for all cells
## Create empty chi_squared_table for later storage
chi_squared_table <- matrix(NA, nrow = 2, ncol = 2,
                            dimnames = list(c("mc-sc", "sc-mc"), c("caus", "temp")))

# Loop: Repeat the following commands ...
for (i in 1:2) { # for each of the 2 rows and
  for (j in 1:2) { # for each of the 2 columns:
    observed_freq <- freqs[i, j] # 1. Get the observed frequencies
    expected_freq <- expected_table[i, j] # 2. Get the expected frequencies
    chi_squared_score <- ((observed_freq - expected_freq)^2) / expected_freq # 3. Compute chi-squared scores
    chi_squared_table[i, j] <- chi_squared_score # 4. Store output in the chi_squared_table
  }
}

chi_squared_table

          caus     temp
mc-sc 17.11278 16.69335
sc-mc 36.76575 35.86463

Or, more elegantly:

freqs_test <- chisq.test(freqs)

# Expected frequencies
freqs_test$expected

       
             caus      temp
  mc-sc 135.79404 139.20596
  sc-mc  63.20596  64.79404

# Chi-squared scores
(freqs_test$residuals)^2

       
            caus     temp
  mc-sc 17.11278 16.69335
  sc-mc 36.76575 35.86463

# Test statistics
freqs_test

    Pearson's Chi-squared test with Yates' continuity correction

data:  freqs
X-squared = 104.24, df = 1, p-value < 2.2e-16

Important

If the data does not meet the (expected) frequency requirements for the \(\chi^2\)-test, the Fisher’s Exact Test is a viable alternative (see ?fisher.test() for details).

6.2 Workflow in R

6.2.1 \(\chi^2\)-test

6.2.1.1 Define hypotheses

• \(H_0:\) The variables ORDER and SUBORDTYPE are independent.

• \(H_1:\) The variables ORDER and SUBORDTYPE are not independent.

6.2.1.2 Descriptive overview

We plot the distribution of clause ORDER depending on SUBORDTYPE. This requires (a) selecting the desired variables, (b) computing the token frequencies and (c) computing the percentages.

# Filter data so as to show only those observations that are relevant

data %>% 
  # Filter columns
  select(ORDER, SUBORDTYPE) %>%
    # Count observations 
    count(ORDER, SUBORDTYPE) %>%  
    # Compute percentages
    mutate(pct = n/sum(n) * 100) -> data_order_subord

knitr::kable(data_order_subord)

ORDER	SUBORDTYPE	n	pct
mc-sc	caus	184	45.657568
mc-sc	temp	91	22.580645
sc-mc	caus	15	3.722084
sc-mc	temp	113	28.039702

Next, we visualise the ORDER distribution using a barplot with a custom y-axis, requiring geom_col().

# Plot distribution

data_order_subord %>%
  # Map variables onto axes
  ggplot(aes(x = ORDER, y = pct, fill = SUBORDTYPE)) +
    # Define plot type
    geom_col(pos = "dodge") +
    # Define theme
    theme_classic()

6.2.1.3 Running the test

# Cross-tabulate the frequencies for the variables of interest

freqs <- table(data$ORDER, data$SUBORDTYPE)

freqs ## Assumption met: all observed freqs => 1

       
        caus temp
  mc-sc  184   91
  sc-mc   15  113

# Run a chis-quared test on the absolute frequencies and print the results

test <- chisq.test(freqs, correct = FALSE)

# Inspect expected frequencies

test$expected # Assumption met: all expected frequences => 5

       
             caus      temp
  mc-sc 135.79404 139.20596
  sc-mc  63.20596  64.79404

6.2.1.4 Optional: Effect size

The sample-size independent effect size measure Cramer’s V (\(\phi\)) is defined as

\[V = \sqrt{\frac{\chi^2}{N \times df}}.\] The outcome varies between \(0\) (= no correlation) and \(1\) (= perfect correlation); cf. also Gries (2013: 186).

# Compute Cramer's V

## By hand:

# Given chi-squared statistic
chi_squared <- unname(test$statistic)

# Total number of observations
total_obs <- sum(freqs)

sqrt(chi_squared / total_obs * (min(dim(freqs)) - 1))

[1] 0.5139168

## Automatically:
library("confintr") # Load library

cramersv(test)

[1] 0.5139168

6.2.1.5 Reporting the results

According to a \(\chi^2\)-test, there is a significant association between clause ORDERand SUBORDTYPE at \(p < 0.001\) (\(\chi^2 = 106.44, df = 1\)), thus justifying the rejection of \(H_0\).

Gries, Stefan Thomas. 2013. Statistics for Linguistics with r: A Practical Introduction. 2nd rev. ed. Berlin; Boston: De Gruyter Mouton.

Heumann, Christian, Michael Schomaker, and Shalabh. 2022. Introduction to Statistics and Data Analysis: With Exercises, Solutions and Applications in r. 2nd ed. Cham: Springer. https://doi.org/10.1007/978-3-031-11833-3.

---

1. The degrees of freedom are calculated by \((\textrm{number of rows} -1) \times (\textrm{number of columns} - 1)\).